On the Evaluation of Dynamic Critiquing: A Large-Scale User Study

Critiquing is an important form of feedback in conversational recommender systems. However, in these systems the user is usually limited to critiquing a single product feature at a time. Recently dynamic critiquing has been proposed to address this shortcoming, by automatically generating compound critiques over multiple features that may be presented to the user at recommendation time. To date a number of different versions of dynamic critiquing have been evaluated in isolation, and with reference to artificial users. In this paper we bring together the main flavors of dynamic critiquing and perform a large-scale comparative evaluation as part of an extensive real-user trial. This evaluation reveals some interesting facts about the way real users interact with critique-based recommenders.